Graphical user interface and data transfer methods in a controlling device

ABSTRACT

A smart device is provided with a main remote control application that may be configured using information. The main remote control application may present images of original remote controls corresponding to devices which are controllable by the configured main remote control application. In connection with a presented image of an original remote control, the display may present icons that are representative of a subset of the buttons of the original remote control. The user interface also allows a user to select amongst the images of the original remote controls to change which appliances are to be controlled via the user interface. A pop-up remote control widget may also be provided which may be invoked without switching to the main remote control application provisioned on the smart device.

RELATED APPLICATION INFORMATION

This application claims the benefit of and is a continuation of U.S.application Ser. No. 13/329,940, filed on Dec. 19, 2011, whichapplication claims the benefit of U.S. Provisional Patent ApplicationNo. 61/442,488 filed Feb. 11, 2011, the disclosures of which areincorporated herein by reference in its entirety.

BACKGROUND

Personal communication, productivity, and entertainment devices such astablet computers, smart phones, portable email devices, e-books,hand-held games and/or game controllers, portable media players, etc.(all referred to hereafter as “smart devices”) are known to includefeatures such as graphical user interfaces on color touch screens,wireless Internet capability, and support for ancillary applications(sometimes referred to as “apps”) such as, for example, calendars,email, maps and navigation, etc. Such ancillary applications may bepre-installed in a smart device or may be made available for download bya user. Certain such apps may comprise an ability to issue commands toentertainment and other appliances, for example to offer the featuresand functionality of a universal remote control, to operate inconjunction with a TV guide display to enable channel selection, etc.Typically such remote control apps may be provisioned with a library ofcommand data and protocols suitable for controlling a plurality ofappliances of different type and/or manufacture (a so-called “universalremote”). Such a command library and the hardware necessary to transmitoperational commands to appliances may be incorporated into or madeavailable to the smart device itself, or may take the form of a separateexternal device which is placed in communication with the smart device(i.e., a relay device or so-called “blaster”). After initialinstallation and configuration of such a smart device app, a GUI may beprovided by means of which a user may be enabled to issue operationalcommands to the configured appliances. Disclosed herein areuser-friendly and convenient GUI methods for facilitating command inputand for transferring configuration data between smart devices.

SUMMARY

The following relates generally to remote control systems and, moreparticularly, to methods for presenting a GUI on a portable device,e.g., a smart device which may incorporate universal remote controlfunctionality to command operation of one or more types of remotelycontrollable appliances of one or more manufacturers. Exemplary smartdevices include but are not limited to Apple's “iPhone” and “iPad,”Google's “Android” platform, Nintendo's “Wii U” controller, Blackberry's“Torch” or “PlayBook,” and various other tablet computers and/or smartphones. Exemplary types of appliances include, but are not limited totelevisions, digital video disc players (DVDs), cable or satellite settop boxes (STBs), digital video recorders (DVRs), audio video (AV)receivers, thermostats, game consoles, lighting systems, etc.

In one described aspect, a smart device remote control application maybe furnished with a touch screen GUI the activatable icons of whichapproximate the layout of the target appliances original equipmentremote control. Since differences in keypad size, form factor, or GUIaesthetics may necessitate that only a portion of the replicated keypadcan be displayed at any one time, a user may be provided with a means toscroll within the GUI display. Separately or in conjunction with thisfeature, user selection of a particular target device to control, e.g.TV, DVD, STB, etc., may be made using a slide-out or pull-down menucomprising images representative of original equipment remote controls.

In another described aspect, provision may be made in a smart deviceremote control app for a quick loading mini control panel comprisingfrequently used control functions, such as for example those associatedwith volume control, which may be invoked by a user withoutnecessitating a loading of the full remote control GUI or interruptionof other currently running apps.

In a yet further described aspect, a remote control app, once installedand configured in a first smart device, may be enabled to display amatrix barcode, such as for example a so-called Quick Response Code or“QR code” containing the remote control apps current configuration data.This displayed matrix barcode may then be photographed, opticallyrecognized, or otherwise captured by a second smart device in order torapidly and conveniently configure a remote control app resident in saidsecond smart device to match the configuration of the remote control appof the first smart device.

A better understanding of the objects, advantages, features, propertiesand relationships of the various aspects described hereinafter will beobtained from the following detailed description and accompanyingdrawings which set forth illustrative embodiments and which areindicative of the various ways in which the principles of the inventionclaimed hereinafter may be employed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the claimed invention, reference may behad to preferred embodiments shown in the attached drawings in which:

FIG. 1 illustrates an exemplary system in which a smart device equippedwith a remote control app constructed according to the description thatfollows may be used;

FIG. 2 illustrates in block diagram form the major components of theexemplary smart device of FIG. 1;

FIG. 3 illustrates an exemplary smart device GUI for use in issuingcommands to controlled appliances;

FIG. 4 illustrates an exemplary smart device GUI for use in selecting anappliance to be controlled;

FIG. 5 illustrates an exemplary pop up mini remote control GUI;

FIG. 6 illustrates exemplary matrix barcodes which may encodeconfiguration information for use by a remote control app of a smartdevice; and

FIG. 7 illustrates exemplary situations in which matrix barcodes such asthose of FIG. 6 may be utilized to transfer information to a remotecontrol app of a smart device.

DETAILED DESCRIPTION

By way of example, FIG. 1 illustrates an exemplary system wherein asmart device 100 may be adapted to issue commands to controllableappliances such as a TV 102, a set top box (STB) 104, an audio/videoreceiver (AVR) 106, a DVD player 108, etc. While illustrated in thecontext of a home entertainment system comprising a TV, STB, AVR and DVDplayer, it is to be understood that controllable appliances may include,but need not be limited to, televisions, VCRs, DVRs, DVD players, cableor satellite converter set-top boxes (“STBs”), amplifiers, AV receivers,CD players, game consoles, home lighting, drapery, fans, HVAC systems,thermostats, personal computers, etc. In the illustrative example ofFIG. 1, a smart device 100 may include both a universal remote controlapp and the necessary hardware to enable direct transmission of commandsto appliances 102 through 108, for example without limitation thetechnology described in co-pending U.S. patent application Ser. No.13/043,915 which is incorporated herein by reference in its entirety.Appliance commands may be issued in the form of infrared signals 110 asillustrated, or in any other suitable format, e.g., via an RF signalsuch as contemplated by RF4CE, Zwave, Bluetooth, etc.; ultrasonicsignal; visible light; etc. as appropriate for the control of eachparticular appliance. In alternative embodiments appliance commands maybe issued indirectly via a proxy, or relay device, which is responsiveto wireless communications received from the universal remote controlapp resident in smart device 100, for example as described in co-pendingU.S. patent application Ser. No. 13/071,661 or 13/328,416, which arealso incorporated herein by reference in their entirety. Since thetechniques and methods described herein are equally applicable to eitherembodiment, these will be described hereafter with reference to thesystem of FIG. 1, it being understood that these may be applied mutatismutandis to other embodiments which may utilize a relay device forissuance of appliance commands.

With reference to FIG. 2, as is known in the art, a smart phone device100 may include as needed for a particular application, processing means200 which may comprise both an application processing section 202 and anRF/DSP processing section 204; an LCD display 206; a keypad 208 whichmay comprise physical keys, touch keys overlaying LCD 206, or acombination thereof; a subscriber identification module (SIM) card 210;memory means 212 which may comprise ROM, RAM, Flash, or any combinationthereof; WiFi and/or Bluetooth wireless interface(s) 214; a wirelesstelephony interface 216; power management circuitry 218 with associatedbattery 220; a USB interface 222 and connector 224; an audio managementsystem 226 with associated microphone 228, speaker 230, and headphonejack 232; optional IR communication means comprising a transmitterand/or receiver 236 with associated IR output diode(s) 234; and variousoptional accessory features 238 such as a digital camera, GPS,accelerometer, etc.

A remote control app resident in such a smart device may draw on alibrary of codes and data suitable for commanding the functionaloperations of various types of appliances of multiple manufacturesand/or models, i.e., a library of remote control codesets for aso-called “universal” controller, as well known in the art. Upon initialinstallation, the remote control app may be provisioned with thespecific command data and protocol (“codeset”) to which each of theto-be-controlled appliances is responsive. As is known in the art, thismay take the form of a pointer or index into a library of codesetspre-stored locally in the memory 212 of smart device 100 or of a relaydevice; individual codesets downloaded to those devices during aconfiguration process; an item-by-item download of individual commandsfrom a codeset on an as-required basis from a library stored locally ona PC or on an appliance such as STB 104 or TV 102, or stored remotely ata headend or internet accessible server; etc. Since methods toaccomplish the identification of suitable commands and/or codesets forcontrolled appliances are well known in the art, for the sake of brevitythese will not be further described herein. Nevertheless, for moreinformation in this regard the interested reader may turn to, forexample, co-pending U.S. patent application Ser. No. 13/198,072,13/198,172, or 13/240,604, all of common ownership and all incorporatedherein by reference in their entirety.

In keeping with the objectives of subject disclosure, identification andprovisioning of appliance command codesets may include a provisioning ofappropriate image elements and data for use in presenting the GUIdisplays described herein. As will become apparent, such image elementsand data may include pictures, simulacrums, graphics, layoutinformation, key position maps, etc. as necessary to create graphicalrepresentations of each appliance's original equipment remote control.Such data may be retrieved together with the command codesets, or may beavailable from an independent source such as a Web service which may beseparately accessed by the smart device remote control app once an exactappliance identity has been established, as appropriate for a particularembodiment.

With reference to FIG. 3, when a smart device remote control app inaccordance with the invention described herein is invoked, it maydisplay a GUI 300 which may be representative of the original equipmentremote control corresponding to the currently selected appliance. Asillustrated, GUI 300 may comprise an area 302 containing touch-activatedicons arranged in a layout which approximates a portion of the OEMremote control keypad. A second area 304 may contain an image 306representative of the entire OEM remote control which may serve toidentify to the user of the app both which appliance is currently undercontrol (STB 104 in the example presented) and, for example by means ofa highlighted area 308, which section of the OEM remote control keypadis currently being displayed in area 302. In order to access otherfunctional areas of the remote control GUI, for example an upper section310 or lower section 312 of the simulated OEM keypad, the remote controlapp may be enabled to recognize vertical sliding motions 314 of a user'sfinger as requests to scroll back and forth within the interface. Insome embodiments this input may be recognized only if performed withinarea 304, while in other embodiments it may recognized when performedanywhere on the touch sensitive surface of smart device 100 (i.e., bothareas 302 and 304 of the illustrated example). In yet furtherembodiments, a finger tap on the desired area of remote control image306 may be used to select the keys to be presented in area 302, eitherin place of as an alternate to the sliding motions described above. Asthe interface scrolls, the highlight area 308 of OEM remote controlimage 306 may adjust accordingly so as to correspond with the OEM remotecontrol section displayed in area 302.

With reference to FIG. 4, in order to permit user selection of differenttarget appliances to be controlled a smart device remote control app maybe enabled to recognize a generally horizontal sliding motion 408 of auser's finger commencing within area 304 as a request to changecontrolled appliance. In response to such a gesture, the remote controlapp may cause display area 304 to expand horizontally as illustrated at304′ in illustrative display 400. The expanded display in area 304′ maycomprise representations of the OEM remote controls of the appliancescurrently configured to be controlled by the remote control app. In theillustrative example these may include image 402 representative ofremote control provided with TV 102, image 404 representative of remotecontrol provided with AVR 106, image 406 representative of remotecontrol provided with DVD player 108, and image 306′ representative of aremote control provided with STB 104. Selection of the appliance to becontrolled may now be conveniently accomplished by way of a finger tapon the image representing a remote control for controlling the desiredappliance. Upon selection of an image, the remote control app mayautomatically return to a state such as illustrated in FIG. 3 with thenewly selected remote control device being displayed. Selection of anarea of the display screen (or provision of other input that is notrecognized as a selection of an image) can also be used to manuallyreturn the remote control app to a state such as illustrated in FIG. 3with the remote control that was displayed upon receiving the generallyhorizontal input 408 again being displayed. Alternatively, the imagespresented for selection can be representations of the appliances thatare to be controlled via the current configuration of the remote controlapp in lieu of or together with the images of the remote controlsprovided with such appliances.

In some embodiments, a smart device remote control app may support auser convenience feature in the form of a pop-up remote control widget,for example as illustrated in FIG. 5. Exemplary widget pop-up GUI 502may comprise a limited set of pre-determined, frequently used appliancecommand functions, for example providing a user direct access to thefive basic functions Home, Volume Up, Volume down, Mute and Play/pauseas illustrated. As will be appreciated, a user of the remote control appmay be required to configure these pop-up key assignments during thesmart device remote control app initial installation process, e.g., byspecifying the particular appliance to which each of the basic commandsis to be directed, for example TV or AVR for volume functions and STB orDVD for play/pause in the illustrative system of FIG. 1. Onceconfigured, widget GUI 502 may be invoked at any point without switchingto the main remote control application, for example by activating anicon provided for that purpose, by a gesture, by motion of the smartdevice such as shaking or flipping, etc., as appropriate for aparticular embodiment. When invoked, the widget GUI 502 may betemporarily superimposed over the current smart device display data 500,for example using a smart device OS feature such the Android operatingsystem “PopupWindow” class. Once the desired appliance adjustments havebeen made, widget GUI 502 may be returned to the idle state (e.g., anon-displayed state) by repeating the action which invoked it, bytouching smart device screen 500 in an area outside of widget display502, etc., as appropriate. In a further embodiment, the command buttonspresented with the widget GUI 502 may be dynamically changed or updatedby the remote control app to present only those command buttons that theremote control app determines are most often selected by a user or aknown user. To this end, the remote control app would then include amonitoring and counting feature to keep track of command selection by auser or known user (to the extent the widget GUI 502 is to bepersonalized).

A displayed matrix barcode may also be utilized to conveniently transferremote control app configuration information from an appliance orcomputer to a smart device, or to transfer configurations between smartdevices. In an illustrative embodiment a standardized matrix barcodesuch as the Denso Wave “Quick Response” or QR code may be utilized. TheQR code format is appropriate for this application since it is scaleableup to a maximum data content of 2,953 bytes and is widely supported bybuilt-in smart device operating systems and image recognition software,both on-board and cloud-based. Since the encoding and decodingtechniques utilized in conjunction with QR matrix codes are standardizedand widely known, these will not be described in further detail herein.Nevertheless, for additional information on this subject the interestedreader may turn to published standards such as for example ISO/IECpublication 18004: “Information technology—Automatic identification anddata capture technique—QR Code 2005 bar code symbology specification.”

By way of example without limitation, QR codes suitable for use inconfiguring a smart device remote control app are illustrated in FIGS. 6and 7, along with a human-readable representation of the data encodedtherein. QR code 602, representative of a matrix barcode which may beused to identify a specific controllable appliance to an app resident insmart device 100′ as illustrated in example 702 of FIG. 7, may comprisea brand, type, model number, and serial field 604; a command code setidentifier 606 (where this is known, if this is not present theapplicable command code set may be inferred from the brand modelinformation 604 as is known in the art, for example by reference to adatabase); territory identification 608 corresponding the geographicarea in which this appliance is marketed (when present, this may be usedfor example to select a language for icon key captions); and in someembodiments, information regarding available appliance input ports 610.

QR code 620, representative of a larger matrix which may be used totransfer an entire equipment configuration between apps resident insmart devices 100 and 100′ as illustrated in example 704 of FIG. 7, orfrom a PC based setup program to a smart device, may comprise a list ofcontrollable appliances 622 which includes type, brand, and codesetidentifiers for each appliance; an activity definition field 624comprising the identities of the devices which are to participate invarious user-defined activity macros; and a favorite channel field 626which may comprise a listing of broadcast channel identifiers andcorresponding channel numbers.

While various concepts have been described in detail, it will beappreciated by those skilled in the art that various modifications andalternatives to those concepts could be developed in light of theoverall teachings of the disclosure. For example, in an alternateembodiment, actual photographic images of OEM remote controls may beused in place of the graphic representations depicted herein. In suchinstances, it will be appreciated that some or all of said photographicimages may be captured by a user of the smart device, for example usinga built-in camera. In other embodiments, images representative of theappliance themselves may be used in place of OEM remote images whenselecting an appliance to be controlled. Furthermore, it will beappreciated that optical transfer of setup and configuration informationto and between smart device remote control apps or other controllingdevices may be accomplished via any convenient standard or proprietaryoptical image encoding system, for example without limitation the EIAData Matrix code, Aztec Code, Siemens Data Matrix, etc.

Further, while described in the context of functional modules andillustrated using block diagram format, it is to be understood that,unless otherwise stated to the contrary, one or more of the describedfunctions and/or features may be integrated in a single physical deviceand/or a software module, or one or more functions and/or features maybe implemented in separate physical devices or software modules. It willalso be appreciated that a detailed discussion of the actualimplementation of each module is not necessary for an enablingunderstanding of the invention. Rather, the actual implementation ofsuch modules would be well within the routine skill of an engineer,given the disclosure herein of the attributes, functionality, andinter-relationship of the various functional modules in the system.Therefore, a person skilled in the art, applying ordinary skill, will beable to practice the invention set forth in the claims without undueexperimentation. It will be additionally appreciated that the particularconcepts disclosed are meant to be illustrative only and not limiting asto the scope of the invention which is to be given the full breadth ofthe appended claims and any equivalents thereof.

All patents cited within this document are hereby incorporated byreference in their entirety.

What is claimed is:
 1. A non-transitory, computer-readable media havingstored thereon instructions which, when executed by a processing deviceof a smart device, perform steps comprising: receiving into the smartdevice data that functions to identify an intended target appliance;configuring a remote control application of the smart device tocommunicate commands to the intended target device identified by thedata received by the smart device, the remote control applicationproviding a main remote control user interface when the remote controlapplication is caused to be invoked on the smart device and a secondaryremote control user interface; wherein the remote control application iscaused to be invoked on the smart device in response to a firstpredetermined input being provided to the smart device and, wheninvoked, provides to a display of the smart device the main remotecontrol user interface wherein the main remote control user interfaceincludes a first plurality of user interface elements whereby userinteractions with the first plurality of user interface elements willcause the smart device to transmit commands for controlling functionaloperations of the intended target device; wherein the secondary remotecontrol user interface is caused to be provided to the display of thesmart device in response to a second predetermined input being providedto the smart device wherein the secondary remote control user interfaceincludes a second plurality of user interface elements whereby userinteractions with the second plurality of user interface elements willcause the smart device to transmit commands for controlling functionaloperations of the intended target device; and wherein the instructionscause the smart device to be responsive to the second predetermined userinput to provide the secondary remote control user interface to thedisplay of the smart device independent of the remote controlapplication being invoked on the smart device.
 2. The non-transitory,computer readable media as recited in claim 1, wherein the secondpredetermined input comprises a predetermined motion gesture made upon asurface of the display of the smart device.
 3. The non-transitory,computer readable media as recited in claim 1, wherein the secondpredetermined input comprises a predetermined movement of the smartdevice.
 4. The non-transitory, computer readable media as recited inclaim 1, wherein the secondary remote control user interface, whenprovided to the display of the smart device, is caused to besuperimposed over content currently being displayed in the display ofthe smart device.
 5. The non-transitory, computer readable media asrecited in claim 1, wherein the secondary remote control user interface,when provided to the display of the smart device, is caused to betemporarily displayed in the display of the smart device.
 6. Thenon-transitory, computer readable media as recited in claim 1, whereinthe second plurality of user interface elements comprises a subset ofand less than all of the first plurality of user interface elements. 7.The non-transitory, computer readable media as recited in claim 1,wherein the instructions cause an infrared transmission system of thesmart device to transmit commands for controlling functional operationsof the intended target device.
 8. The non-transitory, computer readablemedia as recited in claim 1, wherein the instructions cause a radiofrequency transmission system of the smart device to transmit commandsfor controlling functional operations of the intended target device. 9.The non-transitory, computer readable media as recited in claim 1,wherein the smart device comprises a tablet computing device.
 10. Thenon-transitory, computer readable media as recited in claim 1, whereinthe smart device comprises a smart phone.
 11. The non-transitory,computer readable media as recited in claim 1, wherein the data thatfunctions to identify an intended target appliance comprises dataindicative of an appliance type and data indicative of an appliancebrand.
 12. The non-transitory, computer readable media as recited inclaim 11, wherein the instructions cause a menu to be displayed in thedisplay of the smart device, the menu being used to receive the dataindicative of the appliance type and the data indicative of theappliance brand.
 13. The non-transitory, computer readable media asrecited in claim 12, wherein the instructions cause the menu to bedisplayed in the display of smart device in connection with a processthat is used to install the remote control application on the smartdevice.
 14. The non-transitory, computer readable media as recited inclaim 1, wherein an entirety of the display of the smart device is usedby the main remote control user interface and wherein less than theentirety of the display of the smart device is used by the secondaryremote control user interface.
 15. The non-transitory, computer readablemedia as recited in claim 1, wherein the instructions cause the smartdevice to use the data to retrieve from an Internet server a commandcode set appropriate for the intended target device and wherein theretrieved command code set is used when the smart device is caused totransmit commands for controlling functional operations of the intendedtarget device.
 16. The non-transitory, computer readable media asrecited in claim 1, wherein the instructions cause the smart device touse the data to select from a memory device of the smart device acommand code set appropriate for the intended target device and whereinthe selected command code set is used when the smart device is caused totransmit commands for controlling functional operations of the intendedtarget device.
 17. The computer-readable media as recited in claim 6,wherein the second plurality of user interface elements comprises one ormore frequently selected ones of the first plurality of user interfaceelements.
 18. The computer-readable media as recited in claim 17,wherein the instructions monitor selections of the first plurality ofuser interface elements to automatically determine the second pluralityof user interface elements.
 19. The computer-readable media as recitedin claim 1, wherein the second plurality of user interface elementsprovide for control of a predetermined, controllable activity.
 20. Thecomputer-readable media as recited in claim 19, wherein the secondplurality of user interface elements provide for control of volumeoperational functions and media transport functions.